Multi-Dimensional Validation of the Integration of Syntactic and Semantic Distance Measures for Clustering Fibromyalgia Patients in the Rheumatic Monitor Big Data Study.

This study primarily aimed at developing a novel multi-dimensional methodology to discover and validate the optimal number of clusters. The secondary objective was to deploy it for the task of clustering fibromyalgia patients. We present a comprehensive methodology that includes the use of several different clustering algorithms, quality assessment using several syntactic distance measures (the Silhouette Index (SI), Calinski-Harabasz index (CHI), and Davies-Bouldin index (DBI)), stability assessment using the adjusted Rand index (ARI), and the validation of the internal semantic consistency of each clustering option via the performance of multiple clustering iterations after the repeated bagging of the data to select multiple partial data sets. Then, we perform a statistical analysis of the (clinical) semantics of the most stable clustering options using the full data set. Finally, the results are validated through a supervised machine learning (ML) model that classifies the patients back into the discovered clusters and is interpreted by calculating the Shapley additive explanations (SHAP) values of the model. Thus, we refer to our methodology as the clustering, distance measures and iterative statistical and semantic validation (CDI-SSV) methodology. We applied our method to the analysis of a comprehensive data set acquired from 1370 fibromyalgia patients. The results demonstrate that the K-means was highly robust in the syntactic and the internal consistent semantics analysis phases and was therefore followed by a semantic assessment to determine the optimal number of clusters (k), which suggested k = 3 as a more clinically meaningful solution, representing three distinct severity levels. the random forest model validated the results by classification into the discovered clusters with high accuracy (AUC: 0.994; accuracy: 0.946). SHAP analysis emphasized the clinical relevance of "functional problems" in distinguishing the most severe condition. In conclusion, the CDI-SSV methodology offers significant potential for improving the classification of complex patients. Our findings suggest a classification system for different profiles of fibromyalgia patients, which has the potential to improve clinical care, by providing clinical markers for the evidence-based personalized diagnosis, management, and prognosis of fibromyalgia patients.

Distributed application of guideline-based decision support through mobile devices: Implementation and evaluation.

BACKGROUND: Traditionally guideline (GL)-based Decision Support Systems (DSSs) use a centralized infrastructure to generate recommendations to care providers, rather than to patients at home. However, managing patients at home is often preferable, reducing costs and empowering patients. Thus, we wanted to explore an option in which patients, in particular chronic patients, might be assisted by a local DSS, which interacts as needed with the central DSS engine, to manage their disease outside the standard clinical settings. OBJECTIVES: To design, implement, and demonstrate the technical and clinical feasibility of a new architecture for a distributed DSS that provides patients with evidence-based guidance, offered through applications running on the patients' mobile devices, monitoring and reacting to changes in the patient's personal environment, and providing the patients with appropriate GL-based alerts and personalized recommendations; and increase the overall robustness of the distributed application of the GL. METHODS: We have designed and implemented a novel projection-callback (PCB) model, in which small portions of the evidence-based guideline's procedural knowledge are projected from a projection engine within the central DSS server, to a local DSS that resides on each patient's mobile device. The local DSS applies the knowledge using the mobile device's local resources. The GL projections generated by the projection engine are adapted to the patient's previously defined preferences and, implicitly, to the patient's current context, in a manner that is embodied in the projected therapy plans. When appropriate, as defined by a temporal pattern within the projected plan, the local DSS calls back the central DSS, requesting further assistance, possibly another projection. To support the new model, the initial specification of the GL includes two levels: one for the central DSS, and one for the local DSS. We have implemented a distributed GL-based DSS using the projection-callback model within the MobiGuide EU project, which automatically manages chronic patients at home using sensors on the patients and their mobile phone. We assessed the new GL specification process, by specifying two very different, complex GLs: for Gestational Diabetes Mellitus, and for Atrial Fibrillation. Then, we evaluated the new computational architecture by applying the two GLs to the automated clinical management, at real time, of patients in two different countries: Spain and Italy, respectively. RESULTS: The specification using the new projection-callback model was found to be quite feasible. We found significant differences between the distributed versions of the two GLs, suggesting further research directions and possibly additional ways to analyze and characterize GLs. Applying the two GLs to the two patient populations proved highly feasible as well. The mean time between the central and local interactions was quite different for the two GLs: 3.95 ± 1.95 days in the case of the gestational diabetes domain, and 23.80 ± 12.47 days, in the case of the atrial fibrillation domain, probably corresponding to the difference in the distributed specifications of the two GLs. Most of the interaction types were due to projections to the local DSS (83%); others were data notifications, mostly to change context (17%). Some of the data notifications were triggered due to technical errors. The robustness of the distributed architecture was demonstrated through the successful recovery from multiple crashes of the local DSS. CONCLUSIONS: The new projection-callback model has been demonstrated to be feasible, from specification to distributed application. Different GLs might significantly differ, however, in their distributed specification and application characteristics. Distributed medical DSSs can facilitate the remote management of chronic patients by enabling the central DSSs to delegate, in a dynamic fashion, determined by the patient's context, much of the monitoring and treatment management decisions to the mobile device. Patients can be kept in their home environment, while still maintaining, through the projection-callback mechanism, several of the advantages of a central DSS, such as access to the patient's longitudinal record, and to an up-to-date evidence-based GL repository.

A Cyber-Security Risk Assessment Methodology for Medical Imaging Devices: the Radiologists' Perspective.

Medical imaging devices (MIDs) are exposed to cyber-security threats. Currently, a comprehensive, efficient methodology dedicated to MID cyber-security risk assessment is lacking. We propose the Threat identification, ontology-based Likelihood, severity Decomposition, and Risk assessment (TLDR) methodology and demonstrate its feasibility and consistency with existing methodologies, while being more efficient, providing details regarding the severity components, and supporting organizational prioritization and customization. Using our methodology, the impact of 23 MIDs attacks (that were previously identified) was decomposed into six severity aspects. Four Radiology Medical Experts (RMEs) were asked to assess these six aspects for each attack. The TLDR methodology's external consistency was demonstrated by calculating paired T-tests between TLDR severity assessments and those of existing methodologies (and between the respective overall risk assessments, using attack likelihood estimates by four healthcare cyber-security experts); the differences were insignificant, implying externally consistent risk assessment. The TLDR methodology's internal consistency was evaluated by calculating the pairwise Spearman rank correlations between the severity assessments of different groups of two to four RMEs and each of their individual group members, showing that the correlations between the severity rankings, using the TLDR methodology, were significant (P < 0.05), demonstrating that the severity rankings were internally consistent for all groups of RMEs. Using existing methodologies, however, the internal correlations were insignificant for groups of less than four RMEs. Furthermore, compared to standard risk assessment techniques, the TLDR methodology is also sensitive to local radiologists' preferences, supports a greater level of flexibility regarding risk prioritization, and produces more transparent risk assessments.

A dual-layer context-based architecture for the detection of anomalous instructions sent to medical devices.

Complex medical devices are controlled by instructions sent from a host personal computer (PC) to the device. Anomalous instructions can introduce many potentially harmful threats to patients (e.g., radiation overexposure), to physical device components (e.g., manipulation of device motors), or to functionality (e.g., manipulation of medical images). Threats can occur due to cyber-attacks, human error (e.g., using the wrong protocol, or misconfiguring the protocol's parameters by a technician), or host PC software bugs. Thus, anomalous instructions might represent an intentional threat to the patient or to the device, a human error, or simply a non-optimal operation of the device. To protect medical devices, we propose a new dual-layer architecture. The architecture analyzes the instructions sent from the host PC to the physical components of the device, to detect anomalous instructions using two detection layers: (1) an unsupervised context-free (CF) layer that detects anomalies based solely on the instruction's content and inter-correlations; and (2) a supervised context-sensitive (CS) layer that detects anomalies in both the clinical objective and patient contexts using a set of supervised classifiers pre-trained for each specific context. The proposed dual-layer architecture was evaluated in the computed tomography (CT) domain, using 4842 CT instructions that we recorded, including two types of CF anomalous instructions, four types of clinical objective context instructions and four types of patient context instructions. The CF layer was evaluated using 14 unsupervised anomaly detection algorithms. The CS layer was evaluated using six supervised classification algorithms applied to each context (i.e., clinical objective or patient). Adding the second CS supervised layer to the architecture improved the overall anomaly detection performance (by improving the detection of CS anomalous instructions [when they were not also CF anomalous]) from an F1 score baseline of 72.6%, to an improved F1 score of 79.1% to 99.5% (depending on the clinical objective or patient context used). Adding, the semantics-oriented CS layer enables the detection of CS anomalies using the semantics of the device's procedure, which is not possible when using just the purely syntactic CF layer. However, adding the CS layer also introduced a somewhat increased false positive rate (FPR), and thus reduced somewhat the specificity of the overall process. We conclude that by using both the CF and CS layers, a dual-layer architecture can better detect anomalous instructions to medical devices. The increased FPR might be reduced, in the future, through the use of stronger models, and by training them on more data. The improved accuracy, and the potential capability of adding explanations to both layers, might be useful for creating decision support systems for medical device technicians.

Assessment of a personalized and distributed patient guidance system.

OBJECTIVES: The MobiGuide project aimed to establish a ubiquitous, user-friendly, patient-centered mobile decision-support system for patients and for their care providers, based on the continuous application of clinical guidelines and on semantically integrated electronic health records. Patients would be empowered by the system, which would enable them to lead their normal daily lives in their regular environment, while feeling safe, because their health state would be continuously monitored using mobile sensors and self-reporting of symptoms. When conditions occur that require medical attention, patients would be notified as to what they need to do, based on evidence-based guidelines, while their medical team would be informed appropriately, in parallel. We wanted to assess the system's feasibility and potential effects on patients and care providers in two different clinical domains. MATERIALS AND METHODS: We describe MobiGuide's architecture, which embodies these objectives. Our novel methodologies include a ubiquitous architecture, encompassing a knowledge elicitation process for parallel coordinated workflows for patients and care providers; the customization of computer-interpretable guidelines (CIGs) by secondary contexts affecting remote management and distributed decision-making; a mechanism for episodic, on demand projection of the relevant portions of CIGs from a centralized, backend decision-support system (DSS), to a local, mobile DSS, which continuously delivers the actual recommendations to the patient; shared decision-making that embodies patient preferences; semantic data integration; and patient and care provider notification services. MobiGuide has been implemented and assessed in a preliminary fashion in two domains: atrial fibrillation (AF), and gestational diabetes Mellitus (GDM). Ten AF patients used the AF MobiGuide system in Italy and 19 GDM patients used the GDM MobiGuide system in Spain. The evaluation of the MobiGuide system focused on patient and care providers' compliance to CIG recommendations and their satisfaction and quality of life. RESULTS: Our evaluation has demonstrated the system's capability for supporting distributed decision-making and its use by patients and clinicians. The results show that compliance of GDM patients to the most important monitoring targets - blood glucose levels (performance of four measurements a day: 0.87±0.11; measurement according to the recommended frequency of every day or twice a week: 0.99±0.03), ketonuria (0.98±0.03), and blood pressure (0.82±0.24) - was high in most GDM patients, while compliance of AF patients to the most important targets was quite high, considering the required ECG measurements (0.65±0.28) and blood-pressure measurements (0.75±1.33). This outcome was viewed by the clinicians as a major potential benefit of the system, and the patients have demonstrated that they are capable of self-monitoring - something that they had not experienced before. In addition, the system caused the clinicians managing the AF patients to change their diagnosis and subsequent treatment for two of the ten AF patients, and caused the clinicians managing the GDM patients to start insulin therapy earlier in two of the 19 patients, based on system's recommendations. Based on the end-of-study questionnaires, the sense of safety that the system has provided to the patients was its greatest asset. Analysis of the patients' quality of life (QoL) questionnaires for the AF patients was inconclusive, because while most patients reported an improvement in their quality of life in the EuroQoL questionnaire, most AF patients reported a deterioration in the AFEQT questionnaire. DISCUSSION: Feasibility and some of the potential benefits of an evidence-based distributed patient-guidance system were demonstrated in both clinical domains. The potential application of MobiGuide to other medical domains is supported by its standards-based patient health record with multiple electronic medical record linking capabilities, generic data insertion methods, generic medical knowledge representation and application methods, and the ability to communicate with a wide range of sensors. Future larger scale evaluations can assess the impact of such a system on clinical outcomes. CONCLUSION: MobiGuide's feasibility was demonstrated by a working prototype for the AF and GDM domains, which is usable by patients and clinicians, achieving high compliance to self-measurement recommendations, while enhancing the satisfaction of patients and care providers.

An architecture for a continuous, user-driven, and data-driven application of clinical guidelines and its evaluation.

OBJECTIVES: Design, implement, and evaluate a new architecture for realistic continuous guideline (GL)-based decision support, based on a series of requirements that we have identified, such as support for continuous care, for multiple task types, and for data-driven and user-driven modes. METHODS: We designed and implemented a new continuous GL-based support architecture, PICARD, which accesses a temporal reasoning engine, and provides several different types of application interfaces. We present the new architecture in detail in the current paper. To evaluate the architecture, we first performed a technical evaluation of the PICARD architecture, using 19 simulated scenarios in the preeclampsia/toxemia domain. We then performed a functional evaluation with the help of two domain experts, by generating patient records that simulate 60 decision points from six clinical guideline-based scenarios, lasting from two days to four weeks. Finally, 36 clinicians made manual decisions in half of the scenarios, and had access to the automated GL-based support in the other half. The measures used in all three experiments were correctness and completeness of the decisions relative to the GL. RESULTS: Mean correctness and completeness in the technical evaluation were 1±0.0 and 0.96±0.03 respectively. The functional evaluation produced only several minor comments from the two experts, mostly regarding the output's style; otherwise the system's recommendations were validated. In the clinically oriented evaluation, the 36 clinicians applied manually approximately 41% of the GL's recommended actions. Completeness increased to approximately 93% when using PICARD. Manual correctness was approximately 94.5%, and remained similar when using PICARD; but while 68% of the manual decisions included correct but redundant actions, only 3% of the actions included in decisions made when using PICARD were redundant. CONCLUSIONS: The PICARD architecture is technically feasible and is functionally valid, and addresses the realistic continuous GL-based application requirements that we have defined; in particular, the requirement for care over significant time frames. The use of the PICARD architecture in the domain we examined resulted in enhanced completeness and in reduction of redundancies, and is potentially beneficial for general GL-based management of chronic patients.

A multiple-scenario assessment of the effect of a continuous-care, guideline-based decision support system on clinicians' compliance to clinical guidelines.

OBJECTIVES: To quantify the effect of a new continuous-care guideline (GL)-application engine, the Picard decision support system (DSS) engine, on the correctness and completeness of clinicians' decisions relative to an established clinical GL, and to assess the clinicians' attitudes towards a specific DSS. METHODS: Thirty-six clinicians, including residents at different training levels and board-certified specialists at an academic OB/GYN department that handles around 15,000 deliveries annually, agreed to evaluate our continuous-care guideline-based DSS and to perform a cross-over assessment of the effects of using our guideline-based DSS. We generated electronic patient records that realistically simulated the longitudinal course of six different clinical scenarios of the preeclampsia/eclampsia/toxemia (PET) GL, encompassing 60 different decision points in total. Each clinician managed three scenarios manually without the Picard DSS engine (Non-DSS mode) and three scenarios when assisted by the Picard DSS engine (DSS mode). The main measures in both modes were correctness and completeness of actions relative to the PET GL. Correctness was further decomposed into necessary and redundant actions, relative to the guideline and the actual patient data. At the end of the assessment, a questionnaire was administered to the clinicians to assess their perceptions regarding use of the DSS. RESULTS: With respect to completeness, the clinicians applied approximately 41% of the GL's recommended actions in the non-DSS mode. Completeness increased to the performance of approximately 93% of the guideline's recommended actions, when using the DSS mode. With respect to correctness, approximately 94.5% of the clinicians' decisions in the non-DSS mode were correct. However, these included 68% of the actions that were correct but redundant, given the patient's data (e.g., repeating tests that had been performed), and 27% of the actions, which were necessary in the context of the GL and of the given scenario. Only 5.5% of the decisions were definite errors. In the DSS mode, 94% of the clinicians' decisions were correct, which included 3% that were correct but redundant, and 91% of the actions that were correct and necessary in the context of the GL and of the given scenario. Only 6% of the DSS-mode decisions were erroneous. The DSS was assessed by the clinicians as potentially useful. DISCUSSION: Support from the GL-based DSS led to uniformity in the quality of the decisions, regardless of the particular clinician, any particular clinical scenario, any particular decision point, or any decision type within the scenarios. Using the DSS dramatically enhances completeness (i.e., performance of guideline-based recommendations) and seems to prevent the performance of most of the redundant actions, but does not seem to affect the rate of performance of incorrect actions. The redundancy rate is enhanced by similar recent findings in recent studies. Clinicians mostly find this support to be potentially useful for their daily practice. CONCLUSION: A continuous-care GL-based DSS, such as the Picard DSS engine, has the potential to prevent most errors of omission by ensuring uniformly high quality of clinical decision making (relative to a GL-based norm), due to the increased adherence (i.e., completeness) to the GL, and most of the errors of commission that increase therapy costs, by reducing the rate of redundant actions. However, to prevent clinical errors of commission, the DSS needs to be accompanied by additional modules, such as automated control of the quality of the physician's actual actions.

Patient-oriented Computerized Clinical Guidelines for Mobile Decision Support in Gestational Diabetes.

The risks associated with gestational diabetes (GD) can be reduced with an active treatment able to improve glycemic control. Advances in mobile health can provide new patient-centric models for GD to create personalized health care services, increase patient independence and improve patients' self-management capabilities, and potentially improve their treatment compliance. In these models, decision-support functions play an essential role. The telemedicine system MobiGuide provides personalized medical decision support for GD patients that is based on computerized clinical guidelines and adapted to a mobile environment. The patient's access to the system is supported by a smartphone-based application that enhances the efficiency and ease of use of the system. We formalized the GD guideline into a computer-interpretable guideline (CIG). We identified several workflows that provide decision-support functionalities to patients and 4 types of personalized advice to be delivered through a mobile application at home, which is a preliminary step to providing decision-support tools in a telemedicine system: (1) therapy, to help patients to comply with medical prescriptions; (2) monitoring, to help patients to comply with monitoring instructions; (3) clinical assessment, to inform patients about their health conditions; and (4) upcoming events, to deal with patients' personal context or special events. The whole process to specify patient-oriented decision support functionalities ensures that it is based on the knowledge contained in the GD clinical guideline and thus follows evidence-based recommendations but at the same time is patient-oriented, which could enhance clinical outcomes and patients' acceptance of the whole system.

Patient-tailored workflow patterns from clinical practice guidelines recommendations.

MobiGuide is a project devoted to the development of a patient-centric decision support system based on computerized clinical guidelines for chronic illnesses including Atrial Fibrillation (AF). In this paper we describe the process of (1) identifying guideline recommendations that will require patients to take actions (e.g., take measurement, take drug), thus impacting patients' daily-life behavior, (2) eliciting from the medical experts the corresponding set of personalized operationalized advices that are not explicitly written in the guideline (patient-tailored workflow patterns) and (3) delivering this advice to patients. The analysis of the AF guideline has resulted in four types of patient-tailored workflow patterns: therapy-related advisors, measurements advisors, suggestions for dealing with interventions that may require modulating patient therapy, and personalized packages for close monitoring of patients. We will show how these patterns can be generated using information stored in a patient health record that embeds clinical data and data about the patient's personal context and preferences.

[Automatic application of clinical guidelines - from theory to practice].

ClinicaL guidelines (GLs) have been shown to be a powerful tool for enhancing the uniformity and quality of care, reducing its costs. However, since they are typically represented in free text, this leads to low rates of compliance. Therefore, physicians might benefit from GL automated decision support. It should be noted that not many studies evaluate the effect of providing support for the application of GLs over significant stretches of time on the quality of medical decisions. In this paper, we will describe the general architecture of medical decision support systems, review several known GL application frameworks, and focus on the research performed in the medicaL informatics research center at Ben-Gurion University [BGU] of the Negev which developed the Digital ELectronic Guideline Library, called DeGeL. In particular, we will describe a new GL application framework called PICARD that is intended for GL application over time, while ensuring that the GLs recommendations were followed. We will briefly introduce a technical evaluation of PICARD in the cardiology domain to manage patients according to a Coumadin [Warfarin] protocoL, and a functional evaluation in a complex pre-eclampsia/ eclampsia GL in the OB/GYN domain, which we performed with 36 physicians. The results showed that the PICARD creates independence in the quality of the decisions from any particular physician, level of expertise, clinicaL scenario, or decision type within the scenarios. CurrentLy, PICARD is a core component in the EU Mobiguide project, which focuses on remote monitoring and care of chronic patients, using mobile devices to send alerts and recommendations.

Developing nursing computer interpretable guidelines: a feasibility study of heart failure guidelines in homecare.

Homecare is the fastest growing healthcare sector and evidence based information systems are critically needed. Nurses provide most of the care in homecare setting, yet there is a lack of knowledge on the feasibility of applying existing methodologies to generate computer interpretable nursing guidelines for home care. This study examined the feasibility of encoding homecare nursing heart failure guideline into a computer interpretable format. First, we achieved experts' consensus on the relevant guideline. Then, after training on the graphical tool for gradual knowledge specification (Gesher), we generated a comprehensive, hierarchical and time-oriented computer interpretable guideline using one of the guideline modeling languages (Asbru). The final guideline included 167 recommendations and experts' evaluation confirmed the adequacy of guideline knowledge representation. Future work should expand the applicability of our methodology and tools to nursing specialties other than heart failure and develop methods for comprehensive quality evaluation of the resulting guidelines.

A scalable architecture for incremental specification and maintenance of procedural and declarative clinical decision-support knowledge.

Clinical guidelines have been shown to improve the quality of medical care and to reduce its costs. However, most guidelines exist in a free-text representation and, without automation, are not sufficiently accessible to clinicians at the point of care. A prerequisite for automated guideline application is a machine-comprehensible representation of the guidelines. In this study, we designed and implemented a scalable architecture to support medical experts and knowledge engineers in specifying and maintaining the procedural and declarative aspects of clinical guideline knowledge, resulting in a machine comprehensible representation. The new framework significantly extends our previous work on the Digital electronic Guidelines Library (DeGeL) The current study designed and implemented a graphical framework for specification of declarative and procedural clinical knowledge, Gesher. We performed three different experiments to evaluate the functionality and usability of the major aspects of the new framework: Specification of procedural clinical knowledge, specification of declarative clinical knowledge, and exploration of a given clinical guideline. The subjects included clinicians and knowledge engineers (overall, 27 participants). The evaluations indicated high levels of completeness and correctness of the guideline specification process by both the clinicians and the knowledge engineers, although the best results, in the case of declarative-knowledge specification, were achieved by teams including a clinician and a knowledge engineer. The usability scores were high as well, although the clinicians' assessment was significantly lower than the assessment of the knowledge engineers.

Ability of expert physicians to structure clinical guidelines: reality versus perception.

RATIONALE, AIMS AND OBJECTIVES: Structuring Textual Clinical Guidelines (GLs) into a formal representation is a necessary prerequisite for supporting their automated application. We had developed a collaborative guideline-structuring methodology that involves expert physicians, clinical editors and knowledge engineers, to produce a machine-comprehensible representation for automated support of evidence-based, guideline-based care. Our goals in the current study were: (1) to investigate the perceptions of the expert physicians and clinical editors as to the relative importance, for the structuring process, of different aspects of the methodology; (2) to assess, for the clinical editors, the inter-correlations among (i) the reported level of understanding of the guideline structuring ontology's (knowledge scheme's) features, (ii) the reported ease of structuring each feature and (iii) the actual objective quality of structuring. METHODS: A clinical consensus regarding the contents of three guidelines was prepared by an expert in the domain of each guideline. For each guideline, two clinical editors independently structured the guideline into a semi-formal representation, using the Asbru guideline ontology's features. The quality of the resulting structuring was assessed quantitatively. Each expert physician was asked which aspects were most useful for formation of the consensus. Each clinical editor filled questionnaires relating to: (1) the level of understanding of the ontology's features (before the structuring process); (2) the usefulness of various aspects in the structuring process (after the structuring process); (3) the ease of structuring each ontological feature (after the structuring process). Subjective reports were compared with objective quantitative measures of structuring correctness. RESULTS: Expert physicians considered having medical expertise and understanding the ontological features as the aspects most useful for creation of a consensus. Clinical editors considered understanding the ontological features and the use of the structuring tools as the aspects most useful for structuring guidelines. There was a positive correlation (R = 0.87, P < 0.001) between the reported ease of understanding ontological features and the reported ease of structuring those features. However, there was no significant correlation between the reported level of understanding the features - or the reported ease of structuring by using those features - and the objective quality of the structuring of these features in actual guidelines. CONCLUSIONS: Aspects considered important for formation of a clinical consensus differ from those for structuring of guidelines. Understanding the features of a structuring ontology is positively correlated with the reported ease of using these features, but neither of these subjective reports correlated with the actual objective quality of the structuring using these features.

A methodology for evaluation of a markup-based specification of clinical guidelines.

We introduce a three-phase, nine-step methodology for specification of clinical guidelines (GLs) by expert physicians, clinical editors, and knowledge engineers, and for quantitative evaluation of the specification's quality. We applied this methodology to a particular framework for incremental GL structuring (mark-up) and to GLs in three clinical domains with encouraging results.

A quality assessment tool for markup-based clinical guidelines.

We introduce a tool for quality assessment of procedural and declarative knowledge. We developed this tool for evaluating the specification of mark-up-based clinical GLs. Using this graphical tool, the expert physician and knowledge engineer collaborate to perform scoring, using pre-defined scoring scale, each of the knowledge roles of the mark-ups, comparing it to a gold standard. The tool enables scoring the mark-ups simultaneously at different sites by different users at different locations.

DeGeL: a clinical-guidelines library and automated guideline-support tools.

Using machine-interpretable clinical guidelines to support evidence-based medicine promotes the quality of medical care. In this chapter, we present the Digital Electronic Guidelines Library (DeGeL), a comprehensive framework, including a Web-based guideline repository and a suite of tools, to support the use of automated guidelines for medical care, research, and quality assessment. Recently, we have developed a new version (DeGeL.NET) of the digital library and of its different tools. We intend to focus in our exposition on DeGeL's major tools, in particular for guideline specification in a Web-based and stand alone fashion (Uruz and Gesher), tools for search and retrieval (Vaidurya and DeGeLookFor) and for run time application (Spock); and to explain how these tools are combined within the typical lifecycle of a clinical guideline.

A quantitative assessment of a methodology for collaborative specification and evaluation of clinical guidelines.

We introduce a three-phase, nine-step methodology for specification of clinical guidelines (GLs) by expert physicians, clinical editors, and knowledge engineers and for quantitative evaluation of the specification's quality. We applied this methodology to a particular framework for incremental GL structuring (mark-up) and to GLs in three clinical domains. A gold-standard mark-up was created, including 196 plans and subplans, and 326 instances of ontological knowledge roles (KRs). A completeness measure of the acquired knowledge revealed that 97% of the plans and 91% of the KR instances of the GLs were recreated by the clinical editors. A correctness measure often revealed high variability within clinical editor pairs structuring each GL, but for all GLs and clinical editors the specification quality was significantly higher than random (p<0.01). Procedural KRs were more difficult to mark-up than declarative KRs. We conclude that given an ontology-specific consensus, clinical editors with mark-up training can structure GL knowledge with high completeness, whereas the main demand for correct structuring is training in the ontology's semantics.

Runtime application of Hybrid-Asbru clinical guidelines.

Clinical guidelines are a major tool in improving the quality of medical care. However, to support the automation of guideline-based care, several requirements must be filled, such as specification of the guidelines in a machine-interpretable format and a connection to an Electronic Patient Record (EPR). For several different reasons, it is beneficial to convert free-text guidelines gradually, through several intermediate representations, to a machine-interpretable format. It is also realistic to consider the case when an EPR is unavailable. We propose an innovative approach to the runtime application of intermediate-represented Hybrid-Asbru guidelines, with or without an available EPR. The new approach capitalizes on our extensive work on developing the Digital electronic Guideline Library (DeGeL) framework. The new approach was implemented as the Spock system. For evaluation, three guidelines were specified in an intermediate format and were applied to a set of simulated patient records designed to cover prototypical cases. In all cases, the Spock system produced the expected output, and did not produce an unexpected one. Thus, we have demonstrated the capability of the Spock system to apply guidelines encoded in the Hybrid-Asbru intermediate representation, when an EPR is not available.

A graphical framework for specification of clinical guidelines at multiple representation levels.

Formalization of a clinical guideline for purposes of automated application and quality assessment mainly involves conversion of its free-text representation into a machine comprehensible representation, i.e., a formal language, thus enabling auto-mated support. The main issues involved in this process are related to the collaboration between the expert physician and the knowledge engineer. We introduce GESHER - a graphical framework for specification of clinical guidelines at multiple representation levels. The GESHER architecture facilitates incremental specification through a set of views adapted to each representation level, enabling this process to proceed smoothly and in a transparent fashion, fostering extensive collaboration among the various types of users. The GESHER framework supports specification of guidelines at multiple representation levels, in more than one specification language, and uses the DeGeL digital guideline library architecture as its knowledge base. The GESHER architecture also uses a temporal abstraction knowledge base to store its declarative knowledge, and a standard medical-vocabularies server for generic specification of key terms, thus enabling reuse of the specification at multiple sites.

The Digital electronic Guideline Library (DeGeL): a hybrid framework for representation and use of clinical guidelines.

We propose to present a poster (and potentially also a demonstration of the implemented system) summarizing the current state of our work on a hybrid, multiple-format representation of clinical guidelines that facilitates conversion of guidelines from free text to a formal representation. We describe a distributed Web-based architecture (DeGeL) and a set of tools using the hybrid representation. The tools enable performing tasks such as guideline specification, semantic markup, search, retrieval, visualization, eligibility determination, runtime application and retrospective quality assessment. The representation includes four parallel formats: Free text (one or more original sources); semistructured text (labeled by the target guideline-ontology semantic labels); semiformal text (which includes some control specification); and a formal, machine-executable representation. The specification, indexing, search, retrieval, and browsing tools are essentially independent of the ontology chosen for guideline representation, but editing the semi-formal and formal formats requires ontology-specific tools, which we have developed in the case of the Asbru guideline-specification language. The four formats support increasingly sophisticated computational tasks. The hybrid guidelines are stored in a Web-based library. All tools, such as for runtime guideline application or retrospective quality assessment, are designed to operate on all representations. We demonstrate the hybrid framework by providing examples from the semantic markup and search tools.